Issues in Understanding the Scalability of Parallel Systems

Scalability is a term frequently used to qualify the match between an algorithm and architecture in a parallel system (an algorithmarchitecture combination). Evaluating the scalability of a parallel system has widespread applicability. The results from such an evaluation may be used to: select the best architecture platform for an application domain, predict the performance of an application on a larger configuration of an existing architecture, identify application and architectural bottlenecks in a parallel system to suggest application restructuring and architectural enhancements, and glean insight on the interaction between an application and an architecture to understand the scalability of other applicationarchitecture pairs. But evaluating and predicting the scalability of parallel systems poses several problems due to the complex interaction between application characteristics and architectural features. In this paper, we propose an approach for evaluating the scalability of parallel systems and develop a framework for studying the inter-play between applications and architectures. Using this framework, we study the scalability of five parallel scientific applications on shared memory platforms with three different network topologies. We illustrate the power of this framework in addressing two related issues. First, we use it to evaluate abstractions of parallel systems that have been proposed for modeling parallel system behavior. Second, we show its important use in synthesizing architectural requirements from an application perspective. Since real-life applications set the standards for computing, our approach uses such applications for studying the scalability of parallel systems. We call such an application-driven approach a top-down approach to scalability study. The main thrust of this approach is to identify important algorithmic and architectural artifacts that impact the performance of a parallel system, understand the interaction between them, quantify the impact of these artifacts on the execution time of an application, and use these quantifications in studying the scalability of the system. We associate an overhead function with each algorithmic and architectural artifact that impedes the performance of a parallel system. We isolate and quantify the algorithmic overheads such as serial fraction and work-imbalance from the overall execution time of an application. We also isolate other overheads such as network latency (the actual hardware transmission time in the network) and network contention (the amount of time spent waiting for a resource to become free in the network) arising from the interaction of the algorithm with the underlying hardware. Our approach uses a combination of experimentation, simulation and analytical techniques in quantifying these overheads.